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ABSTRACT 

Systemic reform of mathematics and science education in K-12 
schools, sponsored by the National Science Foundation (NSF) , was instituted 
in 1991. Built into the initiatives were guidelines for local (internal) 
evaluation of programs. This paper is a summary of the evaluation 
conceptualizations and efforts of one evaluator at the Fresno Unified School 
District. The evaluator created a model for internal evaluation by combining 
driving elements identified by NSF with process action stages developed by 
the Fresno program to create evaluative views. The model also incorporated 
Stuf f lebeam ' s conceptualization of evaluations for decision-making: context, 
input, process, and production. The model grew into an embedded-research 
approach that includes understanding, describing, and recommending from 
within the district. It also led to the evaluator offering input into the 
decision-making process and to the following key learnings: (1) Evaluation 

design and tools must be flexible and include triangulation, quantitative and 
qualitative methods, and multiple units of analysis; (2) instruments can be 
better developed when evaluators understand the complexities of decision- 
making; (3) evaluations should include close inspections of individual 
program components as well as a means to describe how the program as a whole 
is developing. (WFA) 
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Evaluating a Systemic Reform Project at the School District Level 

National Science Foundation-sponsored systemic reform of mathematics and 
science education in K-12 schools has recently celebrated its 10 th birthday, having been 
officially “bom” with the Statewide Systemic Initiatives (SSI) in 1991. NSF’s systemic 
reform focus has shifted targets through the 90’s, including the provision of funding for 
school districts in urban and rural areas, and more recently on the interactive partnerships 
of schools with institutes of higher education. The study of the efficacy of these systemic 
reform initiatives has occurred at the national level with program evaluations and at the 
local level, with each participating project. NSF’s guidelines for evaluation essentially 
allow for many different orientations to conducting evaluation, provided solid research 
principles are at work (Suter and Frechtling, 1998) 

At the national level, program evaluation efforts such as the one the COSMOS 
Corporation is engaged in are able to see the picture of systemic reform as a patchwork 
quilt of separate projects, albeit each with common themes and characteristics. At the 
local level, the evaluators can view the reform project as an effort to weave a complex 
tapestry of system-wide and system-deep high quality mathematics/science achievement 
for kids. At the root of the task is the need to view efforts as they occur in a system, with 
each component linked to and affecting several other parts of the system. What follows 
here is a summary of the evaluation conceptualizations and efforts of one local evaluator. 

The initial evaluation model in Fresno for the NSF-funded Fresno Systemic 
Program (FSP) in mathematics and science took advantage of Stufflebeam’s 
(Stufflebeam, Foley, Gephart, Guba, Hammond, Merriman, and Provus, 1971) 
conceptualization of four evaluations for decision-making: Context, Input, Process and 
Product (CIPP) evaluations. The FSP evaluation team provided inquiry-activities in these 
four areas to the FSP leadership both as a description of the project status and as an aid 
for making decisions. The emphasis on description has remained, as the evaluation 
methodology has evolved to emphasize descriptive theory via an embedded research 
approach. The fundamental premise of the embedded research view is the immersion of 
the researcher into the system, working with the project to understand and describe what 
is actually happening. 

How does one evaluate the presence and progress of systemic reform in the NSF- 
funded mathematics and science education projects? Returning to the tapestry metaphor, 
evaluating a project’s efforts involves four important elements: a) examining closely 
particular sections of a thread in the tapestry, b) examining the picture produced by the 
thread as it weaves along, c) influencing the tapestry’s design with formative information, 
and d) viewing the picture being woven across the entire tapestry. All four evaluation 
elements involve shifting one’s view of the system; part of the evaluator’s task then is to 
develop an ability to move from view to view. As one noted researcher of systemic 
reform in mathematics and science education has noted, “Evaluating systemic reform 
requires both the capacity of focusing on very detailed student and teacher information 
and pulling back to elucidate equally a global view of individual schools, districts, and 
the whole system,” (Webb, 1999, p. 3). At every level, the evaluator endeavors to 
describe what has happened to bring the system to this place, what is happening now, and 
what may be, given the extension of current variables. 
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Blueprint for Local Evaluation in Fresno: Horizontal and Vertical 
Threads in the Loom 



Fresno Unified School District (student population ~ 79,000) has been engaged in 
an NSF-funded mathematics/science systemic reform effort since 1995-96. The Urban 
Systemic Initiative (USI) and the second cycle of award funding (the Urban Systemic 
Program-USP), have afforded the opportunity for stakeholders to impact student 
achievement through a variety of systemic activities. The NSF has identified some of 
these driving elements of reform, which include: 

• High quality, standards-based curriculum 

• Hands-on, inquiry-based instructional strategies 

• Assessment 

• Professional development for teachers and building administrators 

• Support for students in rigorous coursework 

• Access to rigorous coursework- opportunity to leam 

• Resource development capacity 

• Partnerships with universities and other community stakeholders 

These drivers represent essential areas for projects to address as they engage in impacting 
the system, and are evident across most USIs with NSF award funding (Kim, Crasco, 
Smith, Johnson, Karantonis, and Leavitt, 2001). These elements can be considered to 
comprise our current theory of the content needed for change in the FSP. In tapestry 
language, these are the vertical threads in the loom. 

Threaded horizontally through the driving elements are process action stages. While 
the process may differ for some of the elements, generally each driving element must 
weave its way through these stages of action. These stages also constitute the current 
theory of the process of change as the FSP is conceptualized. 

• Fit with the systemic reform vision 

• Local policy-level decisions 

• Availability and targeting of resources 

• Strategic planning/decisions 

• Implementation 

• Impact 

• Reflection/Feedback to the system 

These interwoven strands also offer a blueprint for evaluation, and the presence and 
impact of a systemic reform effort can be investigated, so long as one keeps in mind the 
four evaluative views: a) close examination of a portion of a thread- a driving element, 
b) looking at the whole picture of the driving element, c) joining in the creation process 
with formative information, and d) stepping back to view the entire picture and its 
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interactions. Figure 1 offers an illustration of the systemic reform metaphor. An 
example of efforts in Fresno to address each of these views is then presented. 



Figure 1: Tapestry of Systemic Reform 
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Close Examination of One thread: Professional Development 
from Implementation to Impact 

The Fresno evaluation team hypothesized that following a single thread could yield 
identifiable impact data; namely that extensive professional development improves the 
quality of instruction, resulting in greater student achievement. Each element in the flow 
chart creates difficulties in operationalization; nevertheless, we hypothesized that data 
can be collected to examine this sequence. 

Professional Development — ► Quality Instruction ► Student Achievement 



Given the hypothesis above, and the limiting factors encountered by the availability 
of research resources, three researchable questions emerged: 

• What connections between the quantity of professional development and the 
quality of instruction can be drawn? 

• What is the correlation between quality instruction and student achievement? 

• What relationship exists between the quantity of professional development 
garnered by teachers and the subsequent achievement of students on available 
assessments? 



Sign-in information from the many professional development offerings were entered into 
a database and used as a measure of the amount of inservice time collected by teachers in 
the system. The quality of instruction could not be measured in any substantial manner; 
however, ratings of individual lessons by trained observers using the Horizon Research 
Classroom Observation Protocol provided a proxy for instructional quality. Student 
achievement was measured using the Stanford Achievement Test, 9 th Edition (SAT-9), 
and the Assessments of Baseline Curriculum Standards (ABCs), developed in 
conjunction with the Northwest Evaluation Association. 



What connections between the quantity of professional development and the 
quality of instruction can be drawn? 

Pearson Correlations were computed between the total 1995-2000 professional 
development hours collected in mathematics or science and the instructional quality 
ratings given to 58 mathematics classrooms between 1998 and 2000. Results indicated 
no apparent relationship (r = -.127, n = 58, p = .343). Several possible explanations for 
the results have occurred, among them: 

• There is no relationship between quantity of professional development and quality 
of instruction. Perhaps data collected on teacher perceived impact of the 
professional development received would relate better to instructional quality. 

• Some other factor (teacher experience?) is related to professional development 
quantity and instructional quality in a more complex relationship (e.g.- 
experienced teachers need less professional development in order to create quality 
instruction than do inexperienced teachers. 

• A relationship exists, but the weakness of the classroom observation instrument in 
assigning a valid year-long rating to instructional quality masks the correlation. 

What is the correlation between quality instruction and student achievement? 

Given that some instructional experiences have been rated highly and others low 
by observers, we next investigated the link between instructional quality and student 
achievement in mathematics. Because not all observed classrooms fell into the Grade 3-6 
range for which two years of test data were available, the number of classrooms included 
in this portion of the study was reduced to 24. Means were initially computed for two 
dependent variables, the ABC pre-post growth score, and the SAT-9 NCE gain score. 
Instructional lessons were categorized into three levels, Ineffective Instruction, Adequate 
Instruction and Effective Instruction. Table 1 summarizes the results. 



Table 1 

Fresno USD Mean ABC and SAT-9 Gains by Classroom Observation Rating 



Instructional Quality Rating 


ABC Growth Mean 


: SAT-9 NCE Gain 


Ineffective Instruction 


7.88 (n = 120) 


0.40 (ti= 187) 


Adequate Instruction 


8.12 (n= 101) 


4.95 (ti= 129) 


Effective Instruction 


8.61 (n= 178) 


5.37 (ti = 209) 
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Initial inspection of the data indicated that as the instructional quality rating ascribed to a 
classroom lesson rose, so did the overall gain scores of students in those classrooms. 
Univariate Analysis of Covariance was performed on the SAT-9 data next, including a 
socio-economic status (SES) indicator (student free-reduced' lunch) and using prior year 
SAT-9 as the covariate. The mathematics instructional quality rating remained a 
significant factor in the model (f= 3.399, p = .034, r 2 = .21 1). 

As with the previous research question, the findings are open to interpretation: 

• There is a link between the observer’s ratings of a classroom lesson and overall 
growth in achievement. 

• The relationship evidenced is not causal. An alternate hypothesis is that 
classrooms receiving higher ratings were populated by students who were 
prepared and eager to learn, making the task of quality instruction easier and 
achievement a logical outcome. 

The small number of students with pre- and post- test scores who were in classrooms of 
an observed teacher calls to question the findings. 

What relationship exists between the quantity of professional development 
garnered bv teachers and the subsequent achievement of students on available 
assessments? 

While the influence of the quantity of professional development on quality 
instruction was not established and the link between instructional quality ratings and 
student growth open to alternative explanations, the third question posed by the 
evaluation team attempted to bypass classroom observations to draw a direct connection 
between professional development and outcomes. Raw and adjusted means are presented 
in Table 2: 



Table 2 

Fresno USD Mean NCE 2000 SAT-9 Scores for Grades 3-6 by Mathematics Professional 
Development Hours Category 1995-2000 . 



Professional 
Development Category 


Mean SAT-9 
NCE 


N= 


Mean SAT-9 Adjusted 
(Fall ’99 ABC pre-score) 


n® 


0 to 29 Hours 


40.5 


3,986 


46.0 


2,095 


30 to 59 Hours 


43.7 


9,740 


48.1 


5,259 


60 to 89 Hours 


46.1 


6,224 


49.4 


3,230 


90+ Hours 


47.5 


4,848 


50.2 


2,831 



Note: Mean differences are significant at/= 33.875, p < .001 , R = .427 



The differences are statistically significant when evaluated with a prior test score 
as the covariate. An initial alternative interpretation is raised with respect to the 
professional development hours categorization. Several hundred new elementary 
teachers have entered the district since the beginning of the FSI in 1995. Many new 
teachers have not had the opportunity to gather the same quantity of professional 
development as more experienced teachers. Differences in the means described above 
may be due to a “teacher experience” factor, by which experienced teachers are able to 
increase student achievement more than new teachers, making the professional 
development quantity variable incidental. In order to study this, all teachers with less 
than 5 years of experience were removed from the data set, and means and Analysis of 
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Covariance run again. Statistically significant differences in the means remained (f = 
15.406, p < .001, r 2 = .446), thus eliminating the “teacher experience” explanation. 

A linear regression analysis was conducted in order to determine the power of 
teacher professional development quantity as a predictive variable. Included as factors in 
the regression were SES, fall pre-score in mathematics, and category of mathematics 
professional development hours. The resulting equation was predictive of the SAT-9 
mathematics score, accounting for 67% of the variance. 



Table 3 

Linear Regression on Fresno USD Grade 3-6 SAT-9 Mathematics NCEs with SES, 
Mathematics Pre-scores and Professional Development Quantity Category as Factors 



Model 


Unstandardized 

Coefficients 


Standardized 

Coefficients 


Significant 


B 


Standard Error 


Beta 


t 


Constant 


-115.27 


1.678 




-68.687 


.000 


Math pre-score 


.800 


.009 


.619 


93.979 


.000 


Prof. Dev. Quan. 


1.165 


.134 


.056 


8.687 


.000 


SES 


2.190 


.105 


.138 


20.921 


.000 



While the model was predictive of mathematics scores on the SAT-9 and the 
contribution of the professional development quantity significant, its contribution to the 
model was not large. 



From Vision to Reflection/Feedback: The Elementary 
Mathematics Professional Development Subsystem 

While the study of the specific effect of mathematics professional development 
quantity has yielded some interesting findings, telescoping out to view this portion of the 
tapestry shows a picture that is full of ambiguities. Evaluation tools that have provided 
the lenses to view this have included: document analysis, formal and informal interviews 
with project staff, surveys of the teaching force, session observations and participant 
post-inservice evaluations. 

Vision 

The vision for elementary mathematics professional development was originally 
established by specifying the quantity and types of inservice that teachers would receive. 
The initial 1995 grant proposal stated that all teachers would receive 20 days, or 120 
hours of professional development in mathematics and science over a five-year period. 
The vision also referenced the need for content and pedagogical professional 
development for teachers. The vision has evolved since the beginning of the project. 
Inspection of the strategic plan beginning in 1998-99 indicates a new emphasis on 
providing differentiated professional development opportunities for teachers to choose 
from as needed. The 2000-05 USP award proposal further develops the notion of 
differentiated professional development according to teacher need, and introduces 
structures for reflection on teacher needs. 
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Policy-related inputs from within and without the FUSD 

Initial policy called for a mandatory five days each year for elementary teachers 
to receive mathematics/science professional development. While implemented through 
1997-98, policies and practices began to shift during year four of the project. Document 
analyses, interviews, and participation/observation at administrative meetings have 
provided the sources for the inputs that have affected the professional development 
system, and are here bulleted: 

• FUSD Policy/Practice decision beginning 1999: no more substitute pull-out days 

• Mandatory mathematics professional development eliminated after 1997-98 

• Reduction of California-funded professional development days from eight (the 
FSP originally mandated five of these to be spent on mathematics and science 
professional development) to three in 2000-01. 

• Change from USI to USP and completion of the Local Systemic Change grant has 
reduced FSP funds from about $4 million annually to $1 million. Other State 
grants have increased this in mathematics, but changes in Eisenhower funding and 
the FUSD budget cuts may mean further reductions as well. 

• Assessment and Accountability initiatives from the State of California have 
increased the anxiety ofi district/school leadership and teachers to increase test 
scores, which may reinforce the belief that a strict focus on skill acquisition is the 
best way to teach. 

• The California Academic Performance Index (API) and the low reading scores in 
the district have raised literacy as a monolithic issue, with the possible result that 
many schools and teachers are focusing on this and minimizing time spent for 
other subjects. 

• A new District-originated “API Alliance” has as a goal the examination and 
selection of a specific tested skill to focus on, with the task of identifying the 
foundation skills needed in order to move students from their current ability to the 
standard. 

• California Content Standards have been established, with specific grade level 
requisite skills. 

• K-12 ,h mathematics adoptions have been undertaken, with committees selecting 
from State-approved lists of texts that align with the standards. 

• Many District-operated professional development initiatives operating 
independently. 

Evaluator interviews with project coordinators offer input on the professional 
development system, it’s impact and its sustainability over the course of the next few 
years. 



• Project Director — 

o We are being affected by a politically driven shift back toward basic skills, 
and away from systemic reform efforts. Big question: How do we address 
sustained efforts at improving mathematics education when the vision for 
this is being constantly buffeted by shifting political winds? 



ERIC 
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• Mathematics Coordinator — 

o Sustainability of the FSP efforts will require that FUSD site leadership 
become/continue to be accountable to teaching for every child. The 
resources and the ‘clout’ that the FSP may have will always be 
changing — but the principal will remain the leader at the site. 

These policies and practices have functioned to strengthen the threads of the professional 
development subsystem in some ways, but to fray them in many other ways. 



Resource Availability and Allocation 

Inspection of the award amounts across the last several years, along with District 
general fund monies and other grants, reveals that relatively less resources are being 
allocated to mathematics professional development in 2000-1 than in previous years. 
Commitments from the general fund have remained stable, but target only salaries for the 
mathematics coordinator and office support. Teachers on Special Assignment (TSAs) 
who provide the bulk of the professional development, have been funded from grant 
awards. The District general fund has been unable to support the quantity of professional 
development evident in the early years of the project. Some grants from the State have 
help to continue inservice support. An interview with the mathematics coordinator 
provided these statements: . 

• A major professional development funder now is the State, with Assembly Bills 
1331, 2442 and 496 grants. These target special populations of teachers for 
specific professional development. 

• New adopted state-recommended curricula meet rigorous content expectations, 
but it is more difficult use conceptually-based pedagogical strategies with it. 



Strategic Planning/Decisions 

Mathematics professional development calendars have been built each spring for 
the following year based on the available funding and the policy changes affecting 
teacher flexibility to attend. As stated earlier, the recent decision to eliminate substitute- 
based full-release inservices has forced the mathematics office to shift the format to 
afternoon and weekend experiences. The FSP staff have increased opportunities for 
teachers to gain content knowledge, assist in curriculum implementation and provide 
lesson design activities through the API Alliance, creating differentiated opportunities. 
However, the number of teachers participating has lessened. This has brought to the 
FSP’s attention a gap in the mathematics professional development implementation: 
many schools and teachers are not developing needs-based professional development 
plans to take advantage of the breadth of offerings. In a 2000-01 teacher survey, only 
17% of teachers reported being involved in planning their mathematics-related 
professional development, and only 23% reported being encouraged to develop an 
individual plan. These numbers represent very little change from the 1998-99 survey. 
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Implementation 

The number of workshops offered and the number of attending teachers has 
dropped off since the'highest participation year of 1997-98. A total of 521 elementary 
teachers attended workshops (some attended multiple workshops) in mathematics and 
science in 2000-01. The 1997-98 school year saw 2,265 teachers receiving professional 
development hours. 

Post-inservice participant evaluation forms provide information about the degree 
of relevancy and overall quality of mathematics inservices as seen by participants. Most 
participants at workshops in 2000-01 (and indeed since 1996) have rated the sessions 
highly. That the workshops have been well-received by participating teachers indicates 
that the professional development system has been effectively meeting teachers’ self- 
reported needs. Table 4 summarizes the post-inservice ratings. 



Table 4 

Participant Evaluations for Mathematics FSP Workshops in 2000-01 (N= 58). 
Rating Scale = 1: None/Not at All. . .5= Extremely/Fully 



Item: 


Number of Sessions with a Mean Rating Between: 
Below 4.0 4.0 and 4.4 Above 4.4 


Meeting content and design provided meaningful 
information/strategies i 


1 


9 


48 


Presentations provided relevance to my needs for 
program implementation 


1 


13 


44 


Presentations provided info ./strategies I will use 


1 


9 


48 


Overall quality of the session 


0 


8 


50 



Impact 

The survey data collected from a representative sample of K-8 th teachers during 4 
years since 1996-97 describes some trend information regarding teacher opinions, 
practices, and professional development experiences that also helps to tell the 
professional development system story. The summary of findings below indicates the 
system’s movement away from attention to investigative practices. 



“ Survey area:' ’ 


^Trends - 


Teacher beliefs about the importance of inquiry-based 
instructional practices 


Initial rises, then drops since 1997-98. 


Teacher preparation to conduct inquiry-based 
instructional practices 


Initial rises, then drops since 1997-98. 


Teacher content proficiency for the grade level taught 


Stable for most content strands, with steady 
increases in pre-algebra and algebra 


Inquiry-based pedagogical strategies 


Slight initial rises, then drops since 1997-98. 


Skill-drill based pedagogical strategies 


Stable, or rising since baseline. 


Prof. Development has increased content knowledge 


Decreasing since 1997-98. 


Prof. Development has increased understanding of 
how children think about mathematics 


Decreasing since 1997-98. 


Prof. Development has increased ability to implement 
high-quality instructional materials 


Decreasing since 1997-98. Preparation to teach 
pre-Algebra and Algebraic concepts has steadily 
risen since 1996. 
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While 2000-01 saw strong effects of State and local policies and practices which 
have seemingly unraveled some of the work of the 1995-2000 USI, a new thread began to 
emerge during the year that contained some of the elements of the original USI vision for 
instruction and for professional development. A group of schools (the API Alliance 
mentioned above) began examining grade- and school-wide weaknesses in particular 
areas of the SAT9 in an effort to increase scores for the State accountability system. 
Although driven by this narrow approach to improving achievement in a single or two 
skill areas, this set of schools and teachers, aided by mathematics FSP staff, began 
outlining the succession of conceptual steps needed for their students to meet the high 
performance benchmarks set by the state. From the outline, lessons were constructed that 
met the performance-conscious needs of site and central administration, yet paid careful 
homage to the conceptually-based progression of skills that formed the initial FSP vision 
in 1995. 

Reflection/Feedback 

The FSP has implemented several activities to provide inservice presenters with 
opportunities to reflect on their experiences. Inservice-presenter teams conduct a 
debriefing after many sessions, discussing the post-workshop evaluations completed by 
participants and their own impressions of how the session was conducted. In addition, 
beginning in 2001-02, the FSP evaluator has begun conducting periodic focus groups 
with the mathematics professional development team to reflect on the program as a 
whole. Annual feedback from the teacher survey and classroom observations is also 
provided to the team from the evaluator. 



Conclusion 

The FSP has continued to develop and provide professional development that 
participants are finding valuable and relevant to their teaching needs. New offerings have 
focused on Algebra content and pedagogy for K-8 th teachers, and also on providing 
university content coursework for groups of teachers. At the same time, many sources 
have conspired to reduce the impact of the mathematics professional development system 
in Fresno in 2000-01. The lessening impact is no surprise given these effects. 

That the capacity-building efforts of the FSP from 1995-2000 would produce new 
efforts at accomplishing change has also been hypothesized; exactly where this would 
emerge in the tapestry has been unknown. The efforts of the API Alliance in 2000-01 
and this year represent the first indications of this newly evolving effort. 

Policy and practice changes, the California accountability system, and funding 
changes have produced effects that rippled through the system in many areas. Impact 
data demonstrates that the professional development system is engendering change in 
fewer teachers than in previous years. The graphic below illustrates how the professional 
development program has affected and been affected by other forces and elements found 
in the system. The elements above the Mathematics Professional Development (PD) box 
represent strands of the tapestry which have supported the life of the professional 
development system, while the elements below represent threads which have been acting 
to fray the tapestry. Data collected concerning some of the specific indicators follow. 
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Evaluator Involvement in Influencing the Process 

The embedded research model posits research of systemic reform as an activity 
one conducts “with the district’^rather than “on the district,” (Webb, 2000). One of the 
key features of the FSP is the utilization of research personnel from the District’s Office 
of Research, Evaluation & Assessment for the major portion of the evaluation of the 
project. By using an “internal evaluator,” the embedding of the researcher in the system 
is a natural occurrence. From 1995-2000, the evaluation of the FSP primarily 
emphasized descriptive theory, using both qualitative and quantitative data collection 
methods to help describe and explain “what is.” But through evaluation presentations, as 
a facilitator of data interpretation and reflection activities for the administrative staff in 
the district, and by simply ‘being around’ (Cronbach, Ambron, Dombusch, Hess, Homik, 
Phillips Walker & Weiner, 1980), the evaluator soon came to be seen by stakeholders as 
a member of the FSP leadership team as well. 

With this presence and acceptance in the system came the opportunity and 
responsibility to offer input into the decision-making process. Internal evaluation during 
the second funding phase from NSF has enabled further evaluation activities from within, 
and allows access to information about decision-making that complements the survey, 
observation and achievement tools that were used previously (Webb, 2000). Data 
interpretation workshops for schools have grown as a function of the Research, 
Evaluation & Assessment office largely from the model initiated by the FSP evaluation. 

An Example: The Use of Reflection Activities to Facilitate Learning 

One example of the interactive use of evaluation findings to inform the project 
began in 1999, as the evaluation team discovered through observation of both 
professional development workshops and classroom lessons that participant/student 
reflection on learning experiences was occurring infrequently. The finding was 
discovered, analyzed and pondered over during an FSP data interpretation workshop with 
the leadership staff and professional development teams. By modeling the process of 
reflection, and by calling attention to its omission in workshops and in classrooms 
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generally, new efforts were undertaken to help participants synthesize newly learned 
material through informal and formal reflection activities. 

Building an infrastructure for teacher reflection on the learning experience for 
children, as well as on one’s own teaching practices has also become a target for the FSP 
from 2000-05. Lead teachers have been established at many school sites, who can 
function as a peer coach for teachers working on improving their instructional strategies. 
Teachers can invite the lead teacher to observe a classroom lesson, then debrief after the 
lesson. One of the proposed features of the FSP is the development of an action- 
reflection, action research component of professional development. The separate 
infrastructure has been implemented sporadically however, with one summer institute 
course for mathematics teachers in 2001 attracting 10 teachers. 

The Entire Tapestry: Changing Status of the System 

Equipped with information from the professional development strand and the 
other major driving elements of the system, one can study the status of the system as a 
whole unit. A helpful model for describing the states of a reforming system has been 
developed by Cosmos Corporation (Yin, Noboa-Rios, Davis, Castillo and MacTurk, 
2001). The model positions a series of driving reform elements as they move toward 
(and away from) alignment in the reform process. Fresno’s study of each of the elements 
and their movement reveals a complex picture, influenced in part from changes at the 
state level. California’s recently legislated elements, including the assessment and 
accountability system, content standards, narrowed curricular choices, and prepackaged 
professional development programs tied to grants, have all affected the reform-based 
alignment of elements in the Fresno system. As the series of California educational 
forces have become implemented or mandated, Fresno’s driving elements have been 
affected, sometimes in a regressive movement away from previous alignment, and 
sometimes toward redefinition. The FSP has worked at the redefinition and realignment 
process, with the current picture of the tapestry revealing some well-bonded and coherent 
elements, and some fraying areas. 



Final Thoughts 

The systemic reform tapestry being woven in Fresno has encountered significant 
challenges and met with some important successes since 1995. While the kinds of 
instruments for evaluation have remained substantially the same over time, the role of the 
evaluator in describing this system has evolved. A descriptive-analytical approach based 
on the CIPP model has grown into an embedded research approach that includes 
understanding, describing and recommending from within. The model allows a “fit” with 
the FSP project, which is itself changing as the system encounters new resources and 
challenges. Some new evaluation activities have been introduced, such as focus groups. 
These key learnings have been guides in the study of the project in process: 

• Evaluation design and tools must be flexible. 

o Immerse oneself in to the system in order to understand the complex 
interactions that make up decisions. Instruments can be better developed 
with this perspective. 
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o Triangulate, triangulate, triangulate! Generally using multiple data 
sources to support conclusions has functioned to increase evaluator 
confidence in conclusions, and project staff confidence in the evaluator! 
o Using both quantitative and qualitative methods increases the chances of 
viably describing the project and progress, 
o Consider multiple units of analysis. NSF asked projects in 1998 to define 
the unit of change — most projects selected the school as the unit of 
change. However, the actual “unit of change” may be different from 
activity to activity and from day to day. The evaluator’s units of analyses 
should include as many of these as possible: the student, the classroom, 
the school, and the entire project. 

• Build a couple of close inspections of individual threads into the evaluation 
design. 

• Develop a wide-angled lens to see the tapestry and describe how it is taking 
shape. 
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